In order to further improve the precision of navigation signal, band efficiency and the reliability performance for navigation systems, a new signal system adopted Minimum Shift Keying (MSK) with Binary Offset Carrier (BOC) based on Low Density Party Check (LDPC) codes was presented, which was called LDPC-MSK-BOC signal system. The navigation performances of the BOC and MSK-BOC were evaluated based on the parameters of Compass and GPS typical signals, which were scaled with power spectral density, code tracking error, multipath error envelope, bit error rate, anti-narrowband jamming merit factor and anti-matched spectrum jamming merit factor for demodulation processing, anti-narrowband jamming merit factor and anti-matched spectrum jamming merit factor for code tracking processing, and spectral separation coefficient. The theoretical analysis and simulation show that the proposed system has better performance in the field of code tracking precision and anti-multipath compared with BOC signal system under the limited spectrum resource condition. Meanwhile, the signal structure can further improve system reliability and band efficiency.